Inferring informational goals and preferred level of detail of answers based on application being employed by the user

ABSTRACT

A system and method for inferring informational goals and preferred level of details in answers in response to questions posed to computer-based information retrieval or question-answering systems is provided. The system includes a query subsystem that can receive an input query and extrinsic data associated with the query and which can output an answer to the query. The query subsystem accesses an inference model to retrieve conditional probabilities that certain informational goals are present. One application of the system includes determining a user&#39;s likely informational goals and then accessing a knowledge data store to retrieve responsive information. Determining a user&#39;s likely informational goals can include inferring a desired level of detail of answers to the query based on the application being employed by the user at the time the query is submitted. The system includes a natural language processor that parses queries into observable linguistic features and embedded semantic components that can be employed to retrieve the conditional probabilities from the inference model. The inference model is built by employing supervised learning and statistical analysis on a set of queries suitable to be presented to a question-answering system. Such a set of queries can be manipulated to produce different inference models based on demographic and/or localized linguistic data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to co-pending Continuation-in-Part U.S.patent application Ser. No. 10/185,150, filed Jun. 28, 2002, entitled,“SYSTEM AND METHODS FOR INFERRING INFORMATIONAL GOALS AND PREFERREDLEVEL OF DETAIL OF RESULTS IN RESPONSE TO QUESTIONS POSED TO ANAUTOMATED INFORMATION-RETRIEVAL OR QUESTION-ANSWERING SERVICE”, andco-pending divisional U.S. patent application Ser. No. 11/047,491, filedon Jan. 31, 2005, entitled, “SYSTEM AND METHODS FOR INFERRINGINFORMATIONAL GOALS AND PREFERRED LEVEL OF DETAIL OF ANSWERS.”

TECHNICAL FIELD

The present invention generally relates to information retrieval, andmore particularly to predicting high-level informational goals and theappropriate level(s) of detail for an answer from observable linguisticfeatures in queries.

BACKGROUND OF THE INVENTION

As the amount of information available to be retrieved by queries tocomputers increases, and as the type and number of information consumersseeking to retrieve such information increases, it has becomeincreasingly important to understand the informational goals of theinformation consumer who generates a query to retrieve such information.Understanding consumer informational goals can improve the accuracy,efficiency and usefulness of a question-answering service (e.g., searchservice) responding to such queries, which leads to improved informationgathering experiences.

Conventional question-answering (QA) systems may employ traditionalinformation retrieval (IR) methods and/or may employ some form ofnatural language processing (NLP) to parse queries. Such systems mayreturn potentially large lists of documents that contain informationthat may be appropriate responses to a query. Thus, an informationconsumer may have to inspect several relevant and/or irrelevantdocuments to ascertain whether the answer sought has been provided. Suchinspection can increase the amount of time spent looking for an answerand reduce the amount of time spent employing the answer. Thus, theefficiency and value of seeking answers through automated questionanswering systems has been limited.

Data associated with informational goals that may be found in a querymay be ignored by conventional systems. Such conventionally ignored datacan provide clues concerning what the information consumer seeks (e.g.,the type of data an information consumer is seeking, the precision of ananswer sought by a query, and other related information in which theinformation consumer may be interested). Conventional statisticalanalysis, when applied to information consumer queries, may yieldinformation that can be employed to improve the relevance of documentsreturned to an information consumer. But the traditional statisticalinformation retrieval approach, even when employing such shallowstatistical methods can still be associated with a poor experience forthe information consumer employing a question answering system.

Addressing in an automated manner queries posed as questions can be moredifficult than addressing traditional keyword-based queries made tosearch engines. Users presenting well-formed questions (e.g., “What isthe capital of Poland?”, “Who killed Abraham Lincoln?”, “Why does itrain?”) typically have a particularly specific information need andcorresponding expectations for receiving a well-focused answer. Butpeople presenting keyword-based queries to the Internet may expect andtolerate a list of documents containing one or more keywords that appearin the free-text query.

SUMMARY OF THE INVENTION

The following presents a simplified summary of the invention in order toprovide a basic understanding of some aspects of the invention. Thissummary is not an extensive overview of the invention. It is notintended to identify key/critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentsome concepts of the invention in a simplified form as a prelude to themore detailed description that is presented later.

The present invention relates to a system and method for inferringinformational goals and/or the appropriate level(s) of detail for ananswer from a query for information. One application of inferring suchinformational goals is to enhance responses to questions presented toquestion answering systems. The present invention improves informationretrieval for questions by predicting high-level informational goalsand/or the appropriate level(s) of detail from observable linguisticfeatures. The present invention can also be employed in otherapplications that benefit from inferring informational goals (e.g.,marketing, demographics). The system includes a learning method thatproduces an inference model and a run-time system that employs theinference model to facilitate inferring informational goals. Thelearning system employs supervised learning with statistical inferencemethods to produce inference data that can be represented by theinference model. Statistical methods with applicability for buildingmodels include Bayesian structure search and parameter optimization,statistical tree induction, support vector machines (SVMs), and neuralnetwork methods. The system can be employed to predict informationalgoals including, but not limited to (1) the type of informationrequested (e.g., definition of a term, value of an attribute,explanation of an event), (2) the topic and focal point of a question,and (3) the level of detail desired in the answer. The predictedinformational goals can then be employed to produce an answer thatincludes information, rather than simply documents, being produced tothe information consumer.

The learning system can retrieve stored information consumer queriesfrom an input query log. Such queries may have been collected from oneinformation consumer and/or one or more groups of information consumersto facilitate localizing the inference model. The learning systememploys supervised learning with statistical inference methods toanalyze queries for information including, but not limited to,linguistic features and user informational goals. The learning systemcan produce, for example, one or more Bayesian network and/or decisiontrees and/or other statistical classification methodology (e.g.,support-vector machines) related to the information associated with thequeries (e.g., the linguistic features and high-level informationalgoals) that facilitate assigning probabilities to differentinformational goals and which thus facilitate increasing the efficiency,accuracy and usefulness of answers produced in response to queries. Inone example of the present invention, during a learning phase, queries,drawn from a log of interactions of users with a server, are processedby a natural language processor (NLP) system. The NLP system decomposesand parses queries into a set of linguistic distinctions including, butnot limited to, distinctions as parts of speech, logical forms, etc. Thedistinctions are considered together with the high-level informationalgoals obtained from human taggers in the process of building predictivemodels.

The linguistic features that are analyzed during the learning processand/or during the run-time processes can include, but are not limitedto, word-based features, structural features and hybrid linguisticfeatures. The high-level informational goals that are analyzed caninclude, but are not limited to, information need, information coveragewanted by the user, the coverage that an expert would give, the inferredage of the user, the topic of the query, the restrictions of the queryand the focus of the query. The results of analyzing such high-levelinformational goals (e.g., inferring age of user) can thus be employedto facilitate more intelligently and thus more precisely and/oraccurately retrieving and/or displaying information via, for example,guided search and retrieval, post-search filtering and/or composition ofnew text from one or more other text sources.

In one example of the present system, the supervised learning undertakenby the learning system was facilitated by the use of a tagging tool. Thetagging tool was operable to present a tagger involved in the supervisedlearning process with detected linguistic features and to inputinformation from the tagger concerning the detected parts of speech andthe informational goals under consideration. Such input informationcould be employed to manipulate values associated with detectedlinguistic features.

The system further includes a run time system. The run time system caninclude a natural language processor, a query subsystem, a knowledgedata store, an inference engine and an answer generator that cooperateto receive an input (e.g., a query) and associated extrinsic data and toproduce an output that relies, at least in part, on the inference modelproduced by the learning system.

The query subsystem can be employed to receive user input (parsed and/orunparsed) and extrinsic data associated with the user input and toproduce an output. The query subsystem may retrieve content to includein the output from a knowledge data store (e.g., online encyclopedia,online help files). The query subsystem may also produce output that isnot an answer to the query. By way of illustration, a user may input aquery with internal contradictions or a user may input a query that thequery subsystem determines may benefit from re-phrasing. Thus, the querysubsystem may produce one or more suggested queries as an output, ratherthan producing an answer. Such queries may subsequently be employed in aquery-by-example system.

The inference engine can be employed to infer informational goals fromthe user input. The inputs to the inference engine can include, but arenot limited to, the user input (parsed and/or unparsed), extrinsic data,and information retrieved from the inference model. The answer generatormay be employed to produce an answer to a query. The inputs to theanswer generator may include, but are not limited to, the original userinput, extrinsic data and informational goals inferred by the inferenceengine.

Thus, by applying supervised learning with statistical analysis to logsof queries posed to question answering systems, the present inventionfacilitates predicting informational goals like the type of informationrequested, the topic, restriction and focal point of the question andthe level of detail of the answer, which facilitates increasing theaccuracy, efficiency and usefulness of such question answering systems.

To the accomplishment of the foregoing and related ends, certainillustrative aspects of the invention are described herein in connectionwith the following description and the annexed drawings. These aspectsare indicative, however, of but a few of the various ways in which theprinciples of the invention may be employed and the present invention isintended to include all such aspects and their equivalents. Otheradvantages and novel features of the invention may become apparent fromthe following detailed description of the invention when considered inconjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram illustrating a system for inferringinformational goals, including both a learning system and a run timesystem, in accordance with an aspect of the present invention.

FIG. 2 is a schematic block diagram illustrating a run time system forinferring informational goals, in accordance with an aspect of thepresent invention.

FIG. 3 is a schematic block diagram illustrating a learning systememployed in creating and/or updating an inference model that can beemployed by a run time system for inferring informational goals, inaccordance with an aspect of the present invention.

FIG. 4 is a schematic block diagram illustrating a hierarchy of detailof knowledge.

FIG. 5 is a schematic block diagram illustrating inferred informationalgoals being employed by a query system to select content to return inresponse to a query, in accordance with an aspect of the presentinvention.

FIG. 6 is a schematic block diagram illustrating a training systememploying natural language processing, supervised learning withstatistical analysis in accordance with an aspect of the presentinvention.

FIG. 7 is a schematic block diagram illustrating conditionalprobabilities associated with determining a probability distributionover a set of goals associated with a query, in accordance with anaspect of the present invention.

FIG. 8 is a schematic block diagram illustrating a Bayesian networkassociated with determining inferences, in accordance with an aspect ofthe present invention.

FIG. 9 is a simulated screen shot depicting a tagging tool employed insupervising learning, in accordance with an aspect of the presentinvention.

FIG. 10 is a flow chart illustrating a method for applying supervisedlearning and Bayesian statistical analysis to produce an inferencemodel, in accordance with an aspect of the present invention.

FIG. 11 is a flow chart illustrating a method for producing a responseto a query where the response benefits from predicted informationalgoals retrieved from a decision model, in accordance with an aspect ofthe present invention.

FIG. 12 is a schematic block diagram of an exemplary operatingenvironment for a system configured in accordance with the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is now described with reference to the drawings,where like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. It may be evident, however, thatthe present invention may be practiced without these specific details.In other instances, well-known structures and devices are shown in blockdiagram form in order to facilitate description of the presentinvention.

As used in this application, the term “component” is intended to referto a computer-related entity, either hardware, a combination of hardwareand software, software, or software in execution. For example, acomponent may be, but is not limited to, a process running on aprocessor, a processor, an object, an executable, a thread of execution,a program, and a computer. By way of illustration, both an applicationrunning on a server and the server can be a component.

As used in this application, the term “engine” is intended to refer to acomputer-related entity, either hardware, a combination of hardware andsoftware, software, or software in execution. For example, an engine maybe, but is not limited to, a process running on a processor, aprocessor, an object, an executable, a thread of execution, a program,and a computer.

As used in this application, the term “natural language processor” issimilarly intended to refer to a computer-related entity, eitherhardware, a combination of hardware and software, software, or softwarein execution.

As used in this application, the term “data store” is intended to referto computer-related data storage. Such storage can include both thehardware upon which physical bits are stored and software datastructures where logical representations are stored. A data store canbe, for example, one or more databases, one or more files, one or morearrays, one or more objects, one or more structures, one or more linkedlists, one or more heaps, one or more stacks and/or one or more datacubes. Furthermore, the data store may be located on one physical deviceand/or may be distributed between multiple physical devices. Similarly,the data store may be contained in one logical device and/or may bedistributed between multiple logical devices.

It is to be appreciated that various aspects of the present inventionmay employ technologies associated with facilitating unconstrainedoptimization and/or minimization of error costs. Thus, non-lineartraining systems/methodologies (e.g., back propagation, Bayesianlearning, decision trees, non-linear regression, or other neuralnetworking paradigms and other statistical methods) may be employed.

Referring initially to FIG. 1, a schematic block diagram illustrates asystem 100 for inferring informational goals in queries, which may beused for enhancing responses to queries presented to an informationretrieval system (e.g., a question answering system). The system 100includes a query subsystem 110 that is employed in processing a userinput 120 and extrinsic data 130 to produce an output 180. The userinput 120 can be, for example, a query presented to a question answeringapplication. The extrinsic data 130 can include, but is not limited to,user data (e.g., applications employed to produce query, device employedto generate query, current content being displayed), context (e.g., timeof day, location from which query was generated, original language ofquery) and prior query interaction behavior (e.g., use of query byexample (QBE), use of query/result feedback). The output 180 mayinclude, but is not limited to, one or more responses, an answerresponsive to a query in the user input 120, one or more re-phrasedqueries, one or more suggested queries (that may be employed, forexample, in a QBE system) and/or an error code.

In an exemplary aspect of the present invention, when the output 180takes the form of one or more responses, the one or more responses maybe further processed to vary in length, precision and detail based, atleast in part, on the inferred informational goals associated with thequery that produced the one or more responses. In another exemplaryaspect of the present invention, the output 180 may be subjected tofurther processing. For example, if the output 180 takes the form of twoor more responses, then the responses may be ranked by a ranking processto indicate, for example, the predicted relevance of the two or moreresponses. Similarly, the output 180 may be further processed by a textfocusing process that may examine the output 180 to facilitate locatingand displaying the piece(s) of information most relevant to the query.Further, the output 180 may be processed, for example, by a diagrammingprocess that displays information graphically, rather than textually.

It is to be appreciated that the term “user” and the term “informationconsumer” contemplate both human and automated query generators. Forexample, a human using a Web browser may generate a query that can beprocessed by the system. Similarly, an information gathering computerapplication may generate a query that can be processed by the system.Thus, the present invention is not intended to be limited to processingqueries produced by humans.

The query subsystem 110 can include an inference engine 112 and aresponse generator 114. The query subsystem 110 can also receive theuser input 120 via a natural language processor 116. The naturallanguage processor 116 can be employed to parse queries in the userinput 120 into parts that can be employed in predicting informationalgoals. The parts may be referred to as “observable linguistic features”.By way of illustration, the natural language processor 116 can parse aquery into parts of speech (e.g., adjectival phrases, adverbial phrases,noun phrases, verb phrases, prepositional phrases) and logical forms.Structural features including, but not limited to, the number ofdistinct parts of speech in a query, whether the main noun in a query issingular/plural, which noun (if any) is a proper noun and the part ofspeech of the head verb post modifier can also be extracted from outputproduced by the natural language processor 116.

The inference engine 112 can be employed to infer informational goals inqueries in the user input 120. The inference engine 112 may access parsedata produced by the natural language processor 116 and an inferencemodel 160 to facilitate inferring such informational goals. For example,the number, type and location of noun phrases and adjectival phrasesdetermined by the natural language processor 116 may be employed toaccess one or more decision trees in the inference model 160 to predictinformational goals. Such high level informational goals can include,but are not limited to, information need, information coverage(s)desired by the user, information coverage that an expert would provide,the inferred age of the user, the topic of the query and the focus ofthe query.

The results of analyzing such high-level informational goals (e.g.,inferring age of user) can thus be employed to facilitate moreintelligently and thus more precisely and/or accurately retrievinginformation via, for example, guided search and retrieval, post-searchfiltering and/or composition of new text from one or more other textsources. By way of illustration, and not limitation, if the age of auser is inferred as being under thirteen, then a first set of resourcesmay be searched and/or a first post-search filter may be employed. Forexample, an encyclopedia designed for children may be searched and aneight letter post-search word size filter may be employed. By way offurther illustration, if the age of a user is inferred as being betweenthirty and forty, then a second set of resources (e.g., regular adultencyclopedia) may be searched and a second post-search filter (e.g.,word sizes up to fourteen letters) may be employed.

The inference engine 112 can also be employed to infer one or morepreferred levels of detail for an answer. Such levels of detail maydepend, for example, in addition to the information retrieved from aquery, on the inferred age of the user, on the physical location of auser (e.g., user on cell phone browser desires less information thanuser on workstation), the organizational location of a user (e.g., CEOdesires different level of detail than mail clerk), one or morerelationships in which the user is engaged (e.g., intern/attending,partner/associate, professor/student) and an application being employedby the user. Once one or more levels of detail are inferred, such levelsof detail may be employed, for example, to determine whether a new textsource should be created from a set of text resources. For example, ifthe level of detail inference indicates that a high-level executivesummary is desired then a summary document may be produced thatincorporates text from one or more other resources. Thus, the executivecan be presented with a single document rather than being forced to wadethrough a number of documents to retrieve the desired information.

By way of illustration and not limitation, the relationships in which auser may be engaged may be employed to infer the level of detail in ananswer. For example, although a user may prefer a short answer to aquestion, a pedagogue may prefer to provide a longer answer as part ofeducating the user. Consider a student presenting a question to a mathprofessor. While the student may simply desire the answer (2x) to thequestion “What is the derivative of x²?”, the teacher may prefer toprovide a more general answer (e.g., the derivative of simple equationsof the form ax^(n) is nax^(n−1)) so that the student may gain more thanjust the answer. Other relationships, and the current context of theparties in the relationship, may similarly facilitate inferring a levelof detail for an answer. For example, while an attending physician mayprovide a first “teaching” answer to an intern treating a non-emergentpatient, the attending physician may provide a second “life-saving”answer to the same intern while treating an emergent patient who is inimminent danger of bleeding out.

The query subsystem 110 can also include a response generator 114. Theresponse generator 114 can, for example, receive as input predictionsconcerning informational goals and can access, for example, theknowledge data store 170 to retrieve information responsive to the queryin the user input 120. The response generator may also produce responsesthat are not answers, but that include rephrased queries and/orsuggested queries. For example, the query subsystem 110 may determinethat the amount and/or type of information sought in a query is so broadand/or voluminous that refining the query is appropriate. Thus, theresponse generator 114 may provide suggestions for refining the query asthe response to the query rather than producing an answer.

The system 100 can also include a learning system 150 that accepts asinputs selected data and/or selected queries from an input query log 140(hereinafter the query log) to produce the inference model 160. Queriesfrom the query log 140 may be passed to the learning system via thenatural language processor 116. The natural language processor 116 canparse queries from the query log 140 into parts that can be employed inlearning to predict informational goals. The parts may also be referredto as “observable linguistic features”.

The query log 140 can be constructed, for example, by gathering queriesfrom the user input 120 (e.g., queries) and/or extrinsic data 130presented to a question answering system. Such queries and/or data maybe collected from one information consumer and/or one or more groups ofinformation consumers. Collecting queries from a heterogeneous groupfacilitates training a system that can be widely used by differentinformation consumers. Collecting queries from a group of homogenousinformation consumers facilitates training a system that can belocalized to that group while collecting queries from a singleinformation consumer facilitates training a system that can becustomized to that individual consumer. In addition to queries, thequery log 140 can contain historical data concerning queries, whichfacilitates analyzing such queries. Further, the query log 140 maycontain additional information associated with actual informationalgoals that can be compared to predicted information goals, with suchactual informational goals employable in supervised learning. In oneexample of the present invention, a query log containing “WH” questions(e.g., who, what, where, when, why, how) and containing an imperative(e.g., name, tell, find, define, describe) was employed. Employing sucha specialized query log 140 can facilitate understanding informationalgoals in different types of queries, which can in turn increase theaccuracy of the response to such queries. Although a query log 140 isdescribed in connection with FIG. 1, it is to be appreciated that thequery log 140 may be manually generated on an ad hoc basis, for example,by a question generator, rather than by collecting queries presented toan actual question answering system. By way of illustration, in alaboratory environment, a linguist interested in applying supervisedlearning to an informational goal inferring system may sit at an inputterminal and generate a thousand queries for input to the learningsystem 150, without ever physically storing the collection of queries.In this illustration, queries from the query log 140 are being consumedas created and may not, therefore, be stored as a complete set in anyphysical device.

The learning system 150 can employ both automated and manual means forperforming supervised learning, with the supervised learning beingemployed to construct and/or adapt data structures including, but notlimited to, decision trees in the inference model 160. Such datastructures can subsequently be employed by the inference engine 112 topredict informational goals in a query in the user input 120. Predictingthe informational goals may enhance the response to a query by returninga precise answer and/or related information rather than returning adocument as is commonly practiced in conventional information retrievalsystems. By way of illustration, the present invention may provideanswers of varying length and level of detail as appropriate to a query.In this manner, an exemplary aspect of the present invention may modelthe expertise of a skilled reference librarian who can not only providethe requested answer but understand the subtleties and nuances in aquestion, and identify an “appropriate” answer to provide to thequerying user. For example, presented with the query “What is thecapital of Poland?” traditional question answering systems may seek tolocate documents containing the terms “capital” and “Poland” and thenreturn one or more documents that contain the terms “capital” and“Poland”. The information consumer may then be forced to read the one ormore documents containing the terms to determine if the answer wasretrieved, and if so, what the answer is. The present invention, byinferring informational goals, identifies conditions under which a moreextended reply, such as “Warsaw is the capital and largest city ofPoland, with a population of approximately 1,700,00” is returned to theuser. The present invention may, for example, set values for severalvariables employed in analyzing the query (e.g., Information Need set to“Attribute”; Topic set to “Poland”; Focus set to “capital”; Cover Wantedset to “Precise”, and Cover Would Give set to “Additional”). Further,the present invention may determine that pictures of landmarks, a citystreet map, weather information and flight information to and fromWarsaw may be included in an appropriate reply. These informationalgoals are predicted by analyzing the observable linguistic featuresfound in the query and retrieving conditional probabilities that certaininformational goals exist from the inference model 160 based on thoseobservable linguistic features. The inference model 160 can beconstructed by employing supervised learning with statistical analysison queries found in one or more query logs 140. The inference model 160can then be employed by a “run time system” to facilitate such enhancedresponses.

In one example of the present invention, the learning system 150 and/orthe inference engine 112 can further be adapted to control and/or guidea dialog that can be employed to clarify information associated withinformational goals, desired level of detail, age and so on. By way ofillustration and not limitation, the learning system 150 may make aninference (e.g., age), but then may present a user interface dialog thatto facilitates clarifying the age of the user. Thus, the learning system150 may be adapted, in-situ, to acquire more accurate informationconcerning inferences, with resulting increases in accuracy. Suchincreased accuracy may be important, for example, in complying withFederal Regulations (e.g., Children's Online Privacy Protection Act). Byway of further illustration, the inference engine 112 may make aninference (e.g., level of detail in answer), but then may present a userinterface that facilitates clarifying the desired level of detail in ananswer. Thus, the inference engine 112 may adapt processes employed ingenerating an inference, and may further adapt search and retrievalprocesses and/or post-search filtering processes to provide a closer fitbetween returned information and desired coverage.

Thus, turning now to FIG. 2, a run time system 200 that can access aninference model 240 to infer informational goals and thus enhanceresponses to queries presented to the run time system 200 isillustrated. The run time system 200 may reside on one computer and/ormay be distributed between two or more computers. Similarly, the runtime system 200 may reside in one process and/or may be distributedbetween two or more processes. Further, the one or more computers and/orone or more processes may employ one or more threads.

The run time system 200 receives data from a user input 220 and may alsoreceive an extrinsic data 230. The user input 220 can include one ormore queries for information. The run time system 200 may receivequeries directly and/or may receive parse data from a natural languageprocessor 216. The queries may appear simple (e.g., what is the deepestlake in Canada?) but may contain informational goals that can beemployed to enhance the response to the query. For example, the query“what is the deepest lake in Canada?” may indicate that the user couldbenefit from receiving a list of the ten deepest lakes in Canada, theten shallowest lakes in Canada, the ten deepest lakes in neighboringcountries, the ten deepest lakes in the world and the ten deepest spotsin the ocean. While there are time and processing costs associated withinferring the informational goals, retrieving the information andpresenting the information to the information consumer, the benefit ofproviding information rather than documents can outweigh that cost,producing an enhanced information gathering experience.

To facilitate enhancing the informational retrieval experience, the runtime system 200 may also examine extrinsic data 230. The extrinsic data230 can include, but is not limited to, user data (e.g., applicationsemployed to produce query, device employed to generate query, currentcontent being displayed), context (e.g., time of day, location fromwhich query was generated, original language of query) and prior queryinteraction behavior (e.g., use of query by example (QBE), use ofquery/result feedback). The user data (e.g., device generating query)can provide information that may be employed in determining what typeand how much information should be retrieved. By way of illustration, ifthe device generating the query is a personal computer, then a firsttype and amount of information may be retrieved and presented, but ifthe device generating the query is a cellular telephone, then a secondtype and amount of information may be retrieved and presented. Thus, theinformational goals of the user may be inferred not only from theobservable linguistic features of a query, but also from extrinsic data230 associated with the query.

The run time system 200 includes a query subsystem 210, which in turnincludes an inference engine 212 and a response generator 214. The querysubsystem 210 accepts parse data produced by a natural languageprocessor 216. The natural language processor 216 takes an input queryand produces parse data including, but not limited to, one or more parsetrees, information concerning the nature of and relationships betweenlinguistic components in the query (e.g., adjectival phrases, adverbialphrases, noun phrases, verb phrases, prepositional phrases), and logicalforms. The query subsystem 210 subsequently extracts structural features(e.g., number of distinct points of speech in a query, whether the mainnoun in a query is singular/plural, which noun (if any) is a proper nounand the part of speech of the head verb post modifier) from the outputof the natural language processor 216. Such parse data can then beemployed by the inference engine 212 to, for example, determine which,if any, of one or more data structures in the inference model 240 toaccess. By way of illustration, first parse data indicating that a firstnumber of nouns are present in a first query may lead the inferenceengine 212 to access a first data structure in the inference model 240while second parse data indicating that a certain head verb postmodifier is present in a second query may lead the inference engine 212to access a second data structure in the inference model 240. By way offurther illustration, the number of nouns and the head verbpost-modifier may guide the initial access to different decision trees(e.g., one for determining information need, one for determining focus)and/or the number of nouns and the head verb post-modifier may guide theaccess to successive sub-trees of the same decision tree.

Based, at least in part, on information retrieved from the inferencemodel 240, the inference engine 212 will determine which, if any,informational goals can be inferred from a query. If one or moreinformational goals can be inferred, then the inference engine 212 mayperform informational goal selection to determine, which, if any,informational goals should be employed by the response generator 214 toproduce a response to the query. By way of illustration, if conflictinginformational goals are inferred from the query, then the inferenceengine 212 may direct the response generator 214 to produce samplequeries that can be employed in subsequent query by example processingby an information consumer. By way of further illustration, if theinference engine 212 determines that a specific informational goal canbe inferred from the query, then the inference engine 212 may direct theresponse generator 214 to retrieve a certain type and/or volume ofinformation that will be responsive to the query and its embeddedinformational goals. Thus, by employing the parse data generated by thenatural language processor 216 and the information stored in theinference model 240, the information gathering experience of aninformation consumer employing the run time system 200 is enhanced ascompared to conventional document retrieval systems.

Thus, referring now to FIG. 3, a training system 300 that can beemployed to create and/or update an inference model 350 is illustrated.The training system 300 may reside on one computer and/or may bedistributed between two or more computers. Similarly, the trainingsystem 300 may reside in one process and/or may be distributed betweentwo or more processes. Further, the one or more computers and/or one ormore processes may employ one or more threads.

The training system 300 includes an input query log 330 that can beconstructed, for example, by gathering queries, parse data, and/orextrinsic data presented to a question answering system. The input querylog 330 can be implemented in, for example, one or more files, one ormore databases, one or more arrays, one or more structures, one or moreobjects, one or more linked lists, one or more heaps and/or one or moredata cubes. The input query log 330 can be stored on one physical deviceand/or can be distributed between two or more physical devices.Similarly, the input query log 330 can reside in one logical datastructure and/or can be distributed between two or more data structures.Furthermore, the input query log 330 may also be implemented in a manualstore. By way of illustration, a technician may manually input queriesto the natural language processor 316, where the queries are fabricatedin the technician's mind at the time of entry. By way of furtherillustration, a technician may manually input queries to the naturallanguage processor 316, where the queries are taken from one or morelists of queries recorded, for example, on paper, or on some separatemagnetic or optical media.

The queries, parse data, and/or extrinsic data can be referred tocollectively as user input 310. The user input 310, via the input querylog 330, can be provided to a learning system 340 via a natural languageprocessor 316. The learning system 340 can include automated and/ormanual means for analyzing the user input 310. Results from analysisperformed on the user input 310 by the learning system 340 can beemployed to create and/or adapt an inference model 350. By way ofillustration, an existing inference model 350 can be adapted by thelearning system 340 based on analyses performed on additional user input310 while a new inference model 350 can be constructed from initialanalyses of user input 310.

The learning system 340 can be employed to compute conditionalprobabilities concerning the likelihood of one or more informationalgoals based on user input 310. The conditional probabilities can beassociated with observable linguistic features and/or relationshipsbetween such linguistic features. The learning system 340 can collectlinguistic data concerning observed linguistic features, and automatedand/or manual means can be employed to manipulate the linguistic data.The linguistic data may indicate, for example, that three nouns arelocated in a query, and a relationship between the number, type andlocation of the nouns.

The linguistic data can then be subjected to statistical analysis tocompute decision trees and/or conditional probabilities based on theobserved linguistic features and the linguistic data. Statisticalmethods of use for building inferential models from this data includeBayesian networks, Bayesian dependency graphs, decision trees, andclassifier models, such as naïve Bayesian classifiers and Support VectorMachines (SVM). Bayesian statistical analysis is known in the art andthus, for the sake of brevity, Bayesian statistical analysis isdiscussed in the context of the analysis applied to observablelinguistic features in accordance with the present invention.

The inference model 350 that can be adapted by the learning system 340can incorporate prior knowledge concerning inferring informationalgoals. For example, an inference model 350 may model prior knowledgethat a first verb (e.g., list, “describe the presidents”) can beemployed to infer that an information consumer desires a broad coveragein the information produced in response to a query while a second verb(e.g., name, “name the 12^(th) president”) can be employed to infer thatan information consumer desires a precise response. Thus, the inferencemodel 350 may start with this prior knowledge and be updated by thelearning system 340. Thus, continued refinements to inference models arefacilitated.

Through the application of Bayes formula the conditional probabilitythat an informational goal can be inferred from the presence and/orabsence of one or more observable linguistic features and/orrelationships between such observable linguistic features isfacilitated. By way of illustration, in one example aspect of thepresent invention, the inferred informational goals accuratelyreflected, at least seventy five percent of the time, actualinformational goals of a consumer offering a question with less thanseven words. By way of further illustration, in another example aspectof the present invention, the inferred informational goals accuratelyreflected, at least sixty percent of the time, actual informationalgoals of a consumer offering a question with seven or more words. Suchaccuracy provides enhanced information retrieval to an informationconsumer. For example, information at an appropriate level ofabstraction may be provided.

While the discussion associated with FIG. 3 deals with inferringinformational goals to enhance answer retrieval, it is to be appreciatedthat such informational goals may be employed in other processes. By wayof illustration, inferred informational goals may be employed inprocesses including, but not limited to, targeting advertising,performing link recommendation, facilitating age determination and/ordemographic modeling. Thus, it is to be appreciated that althoughenhancing answer responsiveness is one application of inferringinformational goals, the present invention is not limited to suchapplication.

Thus, referring now to FIG. 4, a hierarchy 400 of detail/abstraction ofknowledge is illustrated. The hierarchy 400 illustrates that answerswhose return is facilitated by the present invention may include moredetailed information, or more abstract information, based, at least inpart, on informational goals inferred from observable linguisticfeatures in a query. For example, a first query may include observablelinguistic features from which an inference can be made that a precise,detailed answer is desired (e.g., “In which pathways does the proteinPTPμ regulate cell growth?”) while a second query may include observablelinguistic features from which an inference can be made that a moreabstract answer is desired (e.g., How are PTPμ and retinoblastomarelated?). Thus, as compared to a list of documents typically returnedby conventional information retrieval systems, the present inventionfacilitates returning information that may vary in content, length,detail and abstraction level, which improves information retrievalexperiences.

Referring now to FIG. 5, a system 500 for inferring informational goalsbeing employed by a query system 540 to select content to return inresponse to a query 520 is illustrated. A user 510 may present a query520 to the query system 540. The query 520 may be associated with one ormore informational goals 530 _(A1), 530 _(A2) through 530 _(An), n beingan integer (referred to collectively as the informational goals 530).Conventionally, the informational goals 530 may not be retrieved fromthe query 520. But the present invention facilitates inferring the highlevel informational goals 530 from the observable linguistic features inthe query 520.

The high-level informational goals 530 can include, but are not limitedto, information need, information coverage wanted by the user, thecoverage that a query would give, the topic of the query and the focusof the query.

The high-level informational goal referred to as “information need”concerns the type of information requested in a query. Information needtypes can include, but are not limited to, attribute, composition,identification, process, reason, list and target itself. For example, aquery “what is a hurricane?” could have the high-level information goalinformation need established as an “identification” query. Similarly, aquery “where can I find a picture of a horse?” could have the high-levelinformation goal information need established as a “topic itself” query.A topic itself query represents queries seeking access to an objectrather than information concerning an object.

The high-level informational goals referred to as “coverage wanted” and“coverage would give” concern the level of detail to be provided in ananswer to a query. Coverage wanted represents the level of answer detailrequested by the query while coverage would give represents the level ofanswer detail that is as most likely to assist the user in theirinformation quest. Coverage wanted and coverage would give typesinclude, but are not limited to, precise, additional, extended andother. For example, the type “precise” indicates that a query seeks anexact answer (e.g., “who was the 14^(th) president?).

The high-level information goals referred to as “topic” and “focus”concern the linguistic feature representing the topic of discussion in aquery and what the information consumer wants to know about that topicof discussion. While five such informational goals 530 are describedherein (e.g., information need, coverage wanted, coverage would give,topic, focus), it is to be appreciated that a greater or lesser numberof informational goals may be employed in conjunction with the presentinvention.

Based on the high level information goals 530 inferred from theobservable linguistic features in the query 520, one or more contentsources (e.g., content source 550 _(A1), 550 _(A2) through 550 _(An), nbeing an integer, referred to collectively as the content sources 550)may be accessed by the query system 540 to produce a response 560 to thequery 520. The content sources 550 can be, for example, onlineinformation sources (e.g., newspapers, legal information sources, CDbased encyclopedias). Based on the informational goals 530 inferred fromthe query 520, the query system 540 can return information retrievedfrom the content sources 550. Further, the information may vary inaspects including, but not limited to, content, length, scope andabstraction level, for example, again providing an improvement overconventional systems.

Referring now to FIG. 6 a training system 600 including a naturallanguage processor 640, a supervised learning system 660 and a Bayesianstatistical analyzer 662 is illustrated. The training system 600includes a question store 610 as a source of one or more sets ofquestions suitable for posing to a question answering system. Thequestion store 610 maybe a data store and/or a manual store. Thequestion store 610 may be configured to facilitate specific learninggoals (e.g., localization). By way of illustration, questions posed froma certain location (e.g., Ontario) during a period of time (e.g., GreyCup Week) to an online question answering service may be stored in thequestion store 610. The questions may be examined by a question examiner(e.g., linguist, cognitive scientist, statistician, mathematician,computer scientist) to determine question suitability for training, withsome questions being discarded. Further, the questions in the questionstore 610 may be selectively partitioned into subsets including atraining data subset 620 and a test data subset 630. In one exampleaspect of the present invention, questions in the question store 610,the training data 620 and/or the test data 630 may be annotated withadditional information. For example, a linguist may observe linguisticfeatures and annotate a question with such human observed linguisticfeatures to facilitate evaluating the operation of the natural languageprocessor 640. Similarly, a question examiner may annotate a questionwith actual informational goals to facilitate training and/or evaluatingthe operation of the supervised learning system 660 and/or thestatistical analyzer 662.

The training system 600 can present questions and/or annotated data fromthe training data 620 to the natural language processor 640 that canthen observe linguistic features in the question. Such observedlinguistic features can be employed to generate linguistic data that canbe stored in a linguistic data data store 650. For example, thelinguistic data can include, but is not limited to types, numbers and/orlocations of parts of speech (e.g., adjectival phrases, adverbialphrases, noun phrases, verb phrases, prepositional phrases), number ofdistinct parts of speech, existence, location and/or number of propernouns, type of head-verb (e.g., can, be, do, action verbs) and anyhead-verb modifiers (e.g., what, when, where, how adverb, howadjective).

The system 600 can provide linguistic data from the linguistic data datastore 650 to the supervised leaning system 660. Since statisticallearning to build decision trees is employed in the present invention,the supervised learning system 660 may either establish a decision model670 (e.g., a decision tree) and/or update a decision model 670. Thedecision model 670 can store information concerning the likelihood thatcertain informational goals are associated with a question, with theconditional probabilities being computed by the statistical analyzer662. A process and/or human associated with the supervised learningsystem 660 may examine a question, examine the inferences and/orprobabilities associated with the question in the decision model 670 anddetermine that manipulations to the inferences and/or probabilities arerequired. Further, the process and/or human associated with thesupervised learning system 660 may examine a question and the inferencesand/or probabilities associated with the question in the decision model670 and determine that one or more parameters associated with thestatistical analyzer 662 and/or automated processes associated with thesupervised learning system 660 require manipulation. In this manner,different decision models 670 may be produced with biases towardsinferring certain informational goals, which facilitates localizing suchdecision models 670. Such localization can provide improvements overconventional systems.

Referring now to FIG. 7, conditional probabilities associated withanalyzing a probability distribution over a set of goals associated witha query 700 are illustrated. A goal of statistical analysis (e.g.,Bayesian assessment) and inference employed in the present invention isto infer a probability distribution over a set of goals given a query700. The set of probabilities employed in inferring the probabilitydistribution can be accessed by an inference engine 710 to facilitatedetermining which informational goals, if any, should be inferred fromthe probability distribution. Thus, a first conditional probability 720_(A1) (e.g., P(GOAL₁|QUERY)) and a second conditional probability 720_(A2) (e.g., P(GOAL2|QUERY)) through an Nth conditional probability 720_(AN) (e.g., P(GOAL_(N)|QUERY), N being an integer) may be computedthrough, for example, the Bayesian statistical analysis employed in thepresent invention, with such conditional probabilities facilitatingdetermining, by the inference engine 710, whether one or moreinformational goals can be inferred from the query 700. Although thefirst conditional probability 720 _(A1) is illustrated asP(GOAL₁|QUERY), it is to be appreciated that other conditionalprobability statements may be employed in accordance with the presentinvention. For example, a conditional probability P(UIG|QL, POS, PHR,KW, WBF) may be computed where UIG represents a user's informationalgoals, QL represents a Query Length, POS represents a set of Parts ofSpeech encountered in a query, PHR represents a set of phrasesencountered in a query, KW represents a set of keywords encountered in aquery and WBF represents a set of word based features encountered in aquery. It is to be appreciated that other such conditional probabilitiesmay be computed, stored and/or accessed in accordance with the presentinvention.

Referring now to FIG. 8 a Bayesian network 800 associated withdetermining inferences is illustrated. The Bayesian network 800 includesa first term 810 _(A1), and a second term 810 _(A2) through an Nth term810 _(AN), N being an integer (referred to collectively as the terms810). Although N terms are illustrated in FIG. 8, it is to beappreciated that a greater or lesser number of terms may be employed inBayesian networks employed in the present invention. The terms 810 canbe employed in Bayesian inference to infer the likelihood of the goal820 being associated with a query. The measure of the likelihood can becomputed as a conditional probability and stored for access in one ormore decision trees that are included in an inference model, forexample. The Bayesian network 800 indicates that linguistic features canbe employed to infer informational goals in questions. The Bayesiannetwork 800 can be employed in creating and/or adapting an inferencemodel that comprises a probabilistic dependency model.

Linguistic features referred to as “word-based features” can indicatethe presence of one or more specific candidate terms that can beemployed in predicting an informational goal. For example, in oneexample of the present invention, the word “make” was identified as acandidate for determining that an information consumer was interested inthe process for constructing an object and/or in the composition of anobject.

In one example aspect of the present invention, after training had beenperformed, predictions concerning the informational goals InformationNeed, Focus and Cover Wanted could be made based on analyzing thehead-verb pre-modifier (e.g., what, how, who, many, define). Similarly,predictions concerning the informational goal Cover Would Give could bemade based on analyzing the type of head verb (e.g., find, can, be, do,action verb). Further, predictions concerning the informational goalTopic could be made based on the number of nouns in a query andpredictions concerning the informational goal Restriction could be madebased on the part of speech found after the head-verb modifier (e.g.,what is the deepest lake in Canada). It is to be appreciated that theseexample correlations employed in predicting certain informational goalsfrom certain linguistic features and/or relationships between featuresrepresent but one possible set of correlations, and that othercorrelations can be employed in accordance with the present invention.

Referring now to FIG. 9 a simulated screenshot of an exemplary taggingtool 850 employed in supervising learning performed by the presentinvention is presented. The tagging tool 850 includes a query field 860where the query being analyzed is displayed. In the simulatedscreenshot, a natural language processing process has parsed the queryinto parts of speech that are displayed in a parts of speech list 865.The parts of speech list 865 can have identifiers including, but notlimited to, first adjective (ADJ1), second adjective (ADJ2), first verb(VERB1) and second verb (VERB2). In this manner, evaluating theoperation of a natural language processor is facilitated. Further, theparts of speech listed in the parts of speech list 865 facilitateconstructing and analyzing Bayesian networks like that depicted in FIG.8.

The tagging tool 850 also includes a field 870 for displaying thecoverage that a user desires. Information displayed in the field 870 mayrepresent an inference generated by an aspect of the present invention.The tagging tool 850 also includes a field 875 for displaying thecoverage that a tagger employed in supervised learning would assign tothe query. In this way, the tagger can manipulate data and/or valuesgenerated by a parsing and/or tagging process, and can thus affect thecomputed conditional probabilities associated with the informationalgoal being inferred. The tagger may determine to manipulate the data andgenerated by the parsing and/or tagging process to make the data andvalues correspond to a prior schema associated with reasoning concerningthe relevance of a part of a query. Such schemas may vary based ondifferent language models. By way of illustration, if the coverage thatthe tagger infers matches the coverage that an inference enginedetermines that the user wants, then parameters related to machinelearning may be adjusted to affirm the decision made by the inferenceengine to facilitate reaching such a desired decision during futureanalyses. By way of further illustration, if the coverage that thetagger infers does not match the coverage that an inference enginedetermines that the user wants, then parameters related to machinelearning may be adjusted to reduce the likelihood that such an undesireddecision may be made in future analyses.

The tagging tool 850 also includes a topic field 880 that can beemployed to display the part of speech that an aspect of the presentinvention has determined infers the topic of the query. Again, thetagger may manipulate the value represented in the field 880 and thusprovide feedback to the machine learner that will affect futureinferences concerning topic. Similarly, the tagging tool 850 includes afocus field 885 that identifies the part of speech that an aspect of thepresent invention has determined infers the focus of the query. Thetagger may manipulate the value represented in the field 885 and thusprovide feedback to a machine learning process.

The tagging tool 850 also includes buttons that can be employed toprocess the query on a higher level than the field-by-field descriptionprovided above. For example, the tagging tool 850 includes a wrong parsebutton 890 that can be employed to indicate that the parse of the querywas incorrect. Similarly, the tagging tool 850 includes a bad querybutton 895 that can be employed to selectively discard a query, so thatthe analysis of the query is not reflected in an inference model.However, bad queries may optionally be included while non-queries arediscarded, for example.

In view of the exemplary systems shown and described above, amethodology, which may be implemented in accordance with the presentinvention, will be better appreciated with reference to the flowdiagrams of FIGS. 10 and 11. While, for purposes of simplicity ofexplanation, the methodologies are shown and described as a series ofblocks, it is to be understood and appreciated that the presentinvention is not limited by the order of the blocks, as some blocks may,in accordance with the present invention, occur in different ordersand/or concurrently with other blocks from that shown and describedherein. Moreover, not all illustrated blocks may be required toimplement a methodology in accordance with the present invention.

Turning now to FIG. 10, a flow chart illustrates a learning time methodfor applying supervised learning and Bayesian statistical analysis toproduce an inference model. At 900, general initializations occur. Suchinitializations include, but are not limited to, allocating memory,establishing pointers, establishing data communications, acquiringresources, setting variables and displaying process activity. At 910, atraining query is input. At 930 the query is tagged, which can includeparsing the query. For example, different parts of speech may beidentified, and different candidate inferences may be offered bytraining components associated with the present invention. As discussedin association with FIG. 9, such candidate inferences may be manipulatedby a tagger to change the result of the analysis of the query beingtagged. At 940, a decision model can be updated based on the results ofthe analysis and tagging of 930. At 950 a determination is madeconcerning whether more training queries are to be presented to themethod. If the determination at 950 is YES, then processing returns to910 where the next training query is input. If the determination at 950is NO, then processing continues at 995.

At 995, a determination is made concerning whether more testing queriesare going to be presented to the method. If the determination at 995 isNO, then processing concludes. If the determination at 995 is YES, thenprocessing continues at 960 where a testing query is input. At 970,sample output based on the query is produced. The output can include,but is not limited to, an answer to the query, parse data associatedwith the query, linguistic data associated with the query, and potentialupdates to one or more data structures in one or more decision models.At 980, the sample output produced at 970 is analyzed to determineinference accuracy, for example. At 990, based on data and/or statisticsassociated with the analysis of 980 a flag can be set to indicate thatthe decision model should be further updated. For example, an accuracyrate below fifty percent may indicate that further training of thedecision model is warranted or additional factors should be incorporatedin the inference model.

Turning now to FIG. 11, a flow chart illustrates a run time method forproducing a response to a query where the response can benefit frompredicted informational goals retrieved from a decision model. At 1000,general initializations occur. Such initializations include, but are notlimited to, allocating memory, establishing pointers, establishing datacommunications, acquiring resources, setting variables and displayingprocess activity. At 1010 a query is input. The query may be generated,for example, by a human using a browser application or by a computerprocess performing automated information retrieval (e.g., clippingservice). At 1020 the query is parsed into parts and linguistic data isgenerated. At 1030, one or more decision models are accessed using datagenerated by the parsing of 1020. By way of illustration, one or moreparts of speech and/or linguistic data may be employed to select one ormore decision models to access, and those parts of speech may then beemployed to access one or more data structures in the decision model. Byway of further illustration, a first noun may be employed to select afirst decision model, and that first noun may then be employed to begina traverse of a decision tree associated with the first decision modelto retrieve a conditional probability that an informational goal can beinferred from the presence of the noun. For example, the first noun maybe employed to advance the traverse one level in the tree and thenanother linguistic feature (e.g., a type of head verb) can be employedto facilitate advancing to the next level and so on.

At 1040, the one or more conditional probabilities can be examined todetermine which, if any, informational goals can be inferred from thequery. At 1050, based at least in part on the informational goals, ifany, inferred at 1040, the run time method can produce an output. Theoutput may include, but is not limited to, an answer responsive to thenew query, a rephrased query, a query that can be employed in query byexample processing or an error code. At 1060 a determination is madeconcerning whether any more queries are to be presented to the method.If the determination at 1060 is YES, then processing can continue at1010. If the determination at 1060 is NO, then processing can conclude.

In order to provide additional context for various aspects of thepresent invention, FIG. 12 and the following discussion are intended toprovide a brief, general description of one possible suitable computingenvironment 1110 in which the various aspects of the present inventionmay be implemented. It is to be appreciated that the computingenvironment 1110 is but one possible computing environment and is notintended to limit the computing environments with which the presentinvention can be employed. While the invention has been described abovein the general context of computer-executable instructions that may runon one or more computers, it is to be recognized that the invention alsomay be implemented in combination with other program modules and/or as acombination of hardware and software. Generally, program modules includeroutines, programs, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Moreover,one will appreciate that the inventive methods may be practiced withother computer system configurations, including single-processor ormultiprocessor computer systems, minicomputers, mainframe computers, aswell as personal computers, hand-held computing devices,microprocessor-based or programmable consumer electronics, and the like,each of which may be operatively coupled to one or more associateddevices. The illustrated aspects of the invention may also be practicedin distributed computing environments where certain tasks are performedby remote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in both local and remote memory storage devices.

FIG. 12 illustrates one possible hardware configuration to support thesystems and methods described herein. It is to be appreciated thatalthough a standalone architecture is illustrated, that any suitablecomputing environment can be employed in accordance with the presentinvention. For example, computing architectures including, but notlimited to, stand alone, multiprocessor, distributed, client/server,minicomputer, mainframe, supercomputer, digital and analog can beemployed in accordance with the present invention.

With reference to FIG. 12, an exemplary environment 1110 forimplementing various aspects of the invention includes a computer 1112,including a processing unit 1114, a system memory 1116, and a system bus1118 that couples various system components including the system memoryto the processing unit 1114. The processing unit 1114 may be any ofvarious commercially available processors. Dual microprocessors andother multi-processor architectures also can be used as the processingunit 1114.

The system bus 1118 may be any of several types of bus structureincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of commercially available busarchitectures. The computer 1112 memory includes read only memory (ROM)1120 and random access memory (RAM) 1122. A basic input/output system(BIOS), containing the basic routines that help to transfer informationbetween elements within the computer 1112, such as during start-up, isstored in ROM 1120.

The computer 1112 further includes a hard disk drive 1124, a magneticdisk drive 1126, e.g., to read from or write to a removable disk 1128,and an optical disk drive 1130, e.g., for reading a CD-ROM disk 1132 orto read from or write to other optical media. The hard disk drive 1124,magnetic disk drive 1126, and optical disk drive 1130 are connected tothe system bus 1118 by a hard disk drive interface 1134, a magnetic diskdrive interface 1136, and an optical drive interface 1138, respectively.The drives and their associated computer-readable media providenonvolatile storage of data, data structures, computer-executableinstructions, etc. for the computer 1112, including for the storage ofbroadcast programming in a suitable digital format. Although thedescription of computer-readable media above refers to a hard disk, aremovable magnetic disk and a CD, it should be appreciated that othertypes of media which are readable by a computer, such as zip drives,magnetic cassettes, flash memory cards, digital video disks, Bernoullicartridges, and the like, may also be used in the exemplary operatingenvironment, and further that any such media may containcomputer-executable instructions for performing the methods of thepresent invention.

A number of program modules may be stored in the drives and RAM 1122,including an operating system 1140, one or more application programs1142, other program modules 1144, and program non-interrupt data 1146.The operating system 1140 in the illustrated computer can be any of anumber of available operating systems.

A user may enter commands and information into the computer 1112 througha keyboard 1148 and a pointing device, such as a mouse 1150. Other inputdevices (not shown) may include a microphone, an IR remote control, ajoystick, a game pad, a satellite dish, a scanner, or the like. Theseand other input devices are often connected to the processing unit 1114through a serial port interface 1152 that is coupled to the system bus1118, but may be connected by other interfaces, such as a parallel port,a game port, a universal serial bus (“USB”), an IR interface, etc. Amonitor 1154, or other type of display device, is also connected to thesystem bus 1118 via an interface, such as a video adapter 1156. Inaddition to the monitor, a computer typically includes other peripheraloutput devices (not shown), such as speakers, printers etc.

The computer 1112 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remotecomputer(s) 1158. The remote computer(s) 1158 may be a workstation, aserver computer, a router, a personal computer, microprocessor basedentertainment appliance, a peer device or other common network node, andtypically includes many or all of the elements described relative to thecomputer 1112, although, for purposes of brevity, only a memory storagedevice 1160 is illustrated. The logical connections depicted include alocal area network (LAN) 1162 and a wide area network (WAN) 1164. Suchnetworking environments are commonplace in offices, enterprise-widecomputer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 1112 isconnected to the local network 1162 through a network interface oradapter 1166. When used in a WAN networking environment, the computer1112 typically includes a modem 1168, or is connected to acommunications server on the LAN, or has other means for establishingcommunications over the WAN 1164, such as the Internet. The modem 1168,which may be internal or external, is connected to the system bus 1118via the serial port interface 1152. In a networked environment, programmodules depicted relative to the computer 1112, or portions thereof, maybe stored in the remote memory storage device 1160. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

What has been described above includes examples of the presentinvention. It is, of course, not possible to describe every conceivablecombination of components or methodologies for purposes of describingthe present invention, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of the presentinvention are possible. Accordingly, the present invention is intendedto embrace all such alterations, modifications and variations that fallwithin the spirit and scope of the appended claims. Furthermore, to theextent that the term “includes” is used in either the detaileddescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprising” as “comprising” is interpretedwhen employed as a transitional word in a claim.

1. A system for inferring an information goal, comprising: a querysubsystem that receives at least one of a query and an extrinsic data,the query subsystem is operatively coupled to an inference model and aknowledge data store, the query subsystem comprising: a natural languageprocessor that parses the query; and an inference engine that infers oneor more informational goals based, at least in part, on at least one ofthe parsed query, the extrinsic data and an inference data stored in theinference model, the inference engine further inferring one or morepreferred levels of detail for an answer based on an application beingemployed by the user.
 2. The system of claim 1, the informational goalsinclude at least one of, a type of information requested in the query, atopic of the query, or a focal point of the query.
 3. The system ofclaim 2 comprising: an input query log that stores at least one of, oneor more queries or one or more pieces of extrinsic data; and a learningsystem operatively coupled to the input query log, the learning systemoperable to produce the inference model.
 4. The system of claim 3, wherethe learning system comprises: the natural language processor furtherproduces linguistic data concerning one or more linguistic features; atagging tool that facilitates manipulating the linguistic data; one ormore taggers that manipulates the linguistic data; and wherein theinference model stores information concerning conditional probabilitiesassociated with the likelihood that one or more informational goalsexist, where the conditional probabilities of the informational goalsare determined, at least in part, from Bayesian statistical analysisperformed on the linguistic data.
 5. The system of claim 4, thelinguistic data comprises a parse tree, where the parse tree containsextractable information concerning the nature of and relationshipsbetween observable linguistic features.
 6. The system of claim 5, theobservable linguistic features in the extractable information compriseword-based features, structural features and hybrid linguistic features.7. The system of claim 6, the word-based features indicate the presenceof one or more candidate terms that can be employed in predicting aninformational goal.
 8. The system of claim 4, the taggers manipulate thelinguistic data to conform with one or more schemas associated withreasoning concerning the relevance of a part of a query based on one ormore language models.
 9. The system of claim 8, the taggers superviselearning associated with computing probabilities associated with theinformational goals.
 10. The system of claim 4, the inference modelrepresents a probabilistic dependency model.
 11. The system of claim 4,the inference model comprises one or more decision trees, the decisiontrees store conditional probabilities associated with one or moreinformational goals, the decision trees being traversable by thelinguistic data.
 12. The system of claim 3, the input query log is atleast one of a data store or a manual store.
 13. The system of claim 3,the natural language processor parses a query into one or more partssuitable for retrieving one or more conditional probabilities stored inthe inference model.
 14. The system of claim 13, the one or more partscomprise at least one of, logical forms, adjectival phrases, adverbialphrases, noun phrases, verb phrases, prepositional phrases or parsetrees.
 15. The system of claim 14, the inference engine further infersone or more informational goals based, at least in part, on at least oneof the query, the extrinsic data, the one or more parts, or the one ormore conditional probabilities stored in the inference model.
 16. Thesystem of claim 3, the query subsystem further comprising: an answergenerator that produces a response to the query and produces an errormessage.
 17. The system of claim 1, the knowledge data store issearchable for information responsive to a new query and where theinformation retrieved will depend, at least in part, on the inferredinformational goals.
 18. The system of claim 1, the query subsystem iscompiled into an executable, and where the executable accepts as inputone or more query distinctions.
 19. A computer readable medium storingcomputer executable components of a system for inferring an informationgoal, the system comprising: a query component that receives a new queryand a new extrinsic data, the query component operatively coupled to aninference model and a knowledge data store, the query componentcomprising: a natural language processing component that parses the newquery; and an inference component that infers one or more informationalgoals based, at least in part, on at least one of, the new query, thenew extrinsic data and an inference data stored in the inference model,the inference engine further inferring one or more preferred levels ofdetail for an answer based on an application being employed by the user.20. A computer readable medium storing computer executable components ofa system for learning how to infer information goals from queries, thesystem comprising: a natural language processing component that producesa linguistic data concerning one or more linguistic features; a taggingcomponent that manipulates the linguistic data; one or more taggers thatmanipulates the linguistic data; and an inference model component thatstores information concerning conditional probabilities associated withthe likelihood that one or more informational goals exist, where theconditional probabilities of the informational goals are determined, atleast in part, from Bayesian statistical analysis performed on thelinguistic data, the inference engine further inferring one or morepreferred levels of detail for an answer based on an application beingemployed by the user.